[CRITICAL Algorithm] VisionLanguageDataCollator does not accept shift_labels, so this call raises TypeError: __init__() got an unexpected keyword argument 'shift_labels' at runtime — crashing the exact VLM online path this PR adds.

VisionLanguageDataCollator.__init__ (modelopt/torch/utils/plugins/transformers_dataset.py:321) is declared as (self, processor, train_len, chat_template, add_generation_prompt, answer_only_loss, local_image_path, return_labels) — it has neither a shift_labels parameter nor **kwargs, and it never forwards shift_labels to super().__init__. The text-only branch above (LanguageDataCollator, line 135) is fine because that class does declare shift_labels, but the VLM subclass does not.

This isn't caught by test_vlm_data_module_passes_dflash_label_mode because that test replaces VisionLanguageDataCollator with a MagicMock, which accepts any kwargs — so the real signature mismatch is masked.

Impact: DFlash for Qwen3-VL requires shift_labels=False, and passing it is the only way to get the unshifted-label behavior DFlash needs — the crash blocks the feature entirely for any real run.

Fix: add shift_labels to VisionLanguageDataCollator.__init__ and forward it to super().__init__(...) (the parent already stores and uses it). For example, in transformers_dataset.py:

def __init__(
    self,
    processor: str,
    train_len: int = 8192,
    chat_template: str | None = None,
    add_generation_prompt: bool = False,
    answer_only_loss: bool = False,
    shift_labels: bool = True,
    local_image_path: str = "",
    return_labels: bool = False,
):
    ...
    super().__init__(
        tokenizer=self.processor.tokenizer,
        train_len=train_len,
        chat_template=chat_template,
        add_generation_prompt=add_generation_prompt,
        answer_only_loss=answer_only_loss,
        shift_labels=shift_labels,
        return_labels=return_labels,
    )

Consider having the test construct the real collator (or assert against its actual signature) so this class of mismatch is caught.

question: is this patch only needed for transformers 5.3, or for all transformers 5.x or >5.3?

[AI review] The PR description says VisionLanguageDataCollator was extended (propagating answer_only_loss/chat-template/label-alignment settings, VLM_MIN_PIXELS/VLM_MAX_PIXELS limits, ChatML-boundary assistant masks, fixed training_seq_len enforcement), but modelopt/torch/utils/plugins/transformers_dataset.py is not part of this diff — on the current head the class still has its old signature. It looks like that file may not have been committed/pushed.

Note this is broader than the shift_labels TypeError already flagged: even after adding that one parameter, the described mask/pixel/seq-len behaviors would still be missing from the PR.

[AI review] Two robustness concerns with this gate:

1. Version pin. Tracing transformers 5.3.0, the per-video vs per-frame grid mismatch looks like a 5.3 bug rather than a stable contract: get_rope_index consumes one grid row per rendered visual group (next(grid_iters[...])), while the processor renders T timestamp-separated frame groups per video and the video processor emits one [T, H, W] row per video. Even the vanilla generation path (_prepare_position_ids_for_generation) passes the raw grid, so upstream video inference appears equally broken on 5.3 — meaning upstream will likely fix this inside get_rope_index (as in 4.x, where the expansion was internal). If that fix lands in a 5.3.x patch release, this external expansion becomes a double expansion and silently corrupts positions; on 5.4+ the helper silently disables itself and multimodal batches fall back to the broken internal path. Suggest linking the upstream issue/commit here, and failing loudly (frame-group count ≠ grid-row count) instead of silently returning None when the gate doesn't match.

2. model_type != "qwen3_vl" exact match. Family variants like qwen3_vl_moe skip the precomputation but still enter the multimodal top-level forward below — landing exactly in the broken path this helper works around. Consider covering the family or at least warning.

[AI review] This runs before the if not self.training branch, so it also fires on eval prefill (multimodal inputs, no cache). Passing explicit position_ids makes Qwen3VLModel.forward skip compute_3d_position_ids, so self.rope_deltas is never established; a subsequent cached decode step (position_ids=None, past_key_values not None → this helper returns None) then falls into the rope_deltas-based branch with a stale/None delta and computes wrong positions.

Since the docstring says this construction is only needed for DFlash training, consider gating on self.training.

[AI review] base_forward_kwargs forwards everything except assistant_masks/loss_mask. Anything else that reaches forward's **kwargs — e.g. Trainer-injected num_items_in_batch (Trainer detects that this forward accepts **kwargs and adds loss kwargs), or stray dataset columns — gets passed to the HF multimodal forward and can raise TypeError or be silently swallowed depending on version. The text-only branch below is immune because it forwards a fixed argument set.

Consider an allowlist (the multimodal keys + known model kwargs) or filtering against the base forward's signature.

[AI review] nit: import transformers lands between import torch and import torch.nn.functional as F; ruff/isort (I001) will flag this in pre-commit/CI.